The present invention is directed to digital neural networks executed in integrated circuit technology.
Artificial neural networks (referred to below as ANN) are proposed for the parallel processing of extremely large data sets with the goal of pattern recognition and processing (for example, of speech or images). Known ANN of this species are composed of a plurality of non-linear processor elements (neurons) that are networked with one another via variable "weighting factors".
The following non-linear transfer characteristics have been proposed for modeling the neurons, c.f. R. P. Lippmann, "An Introduction to Computing with Neural Nets", IEEE ASSP Magazine, Apr. 1987, pages 4-22:
binary decision, PA1 linear ramp function with saturation characteristic, PA1 sigmoidal function, PA1 tangent-hyperbola function.
Further, a plurality of different structures for the networking of the neurons exist (for example "Hopfield net", "Hamming net", "Perceptron", c.f. R. P. Lippmann as well). A direct digital realization of large nets of this type seems impossible with the current technologies for manufacturing integrated circuits because of the plurality of neurons (&gt;1000) that are required. The main problem is thereby that the number of connections and, thus, the number of variable weighting factors quadratically increases with the number of neurons. For example, 1,000,000 weights are required given 1000 neurons in a completely meshed network.
A few realizations of ANN having, for example, 54 or 256 neurons are disclosed in the literature for a programmable or non-programmable network, respectively, c.f. H. P. Graf, P. De Vegvar, "A CMOS Associative Memory Chip based on Neural Networks, Proc. 1987 IEEE Int. Conf. on Solid State Circuits, pages 304, 305, 437; H.P. Graf et al, "VLSI implementation of a neural network memory with several hundreds of neurons", AIP Conference Proceedings 151, "Neural Networks for Computing", pages 182 through 187, Snowbird, Utah, 1986; W. Hubbard et al, "Electronic Neural Networks", AIP Conference Proceedings 151, "Neural Networks for Computing", pages 227 through 234, Snowbird, Utah, 1986. Proposed realizations have also been disclosed in J. P. Sage, K. Thompson, R. S. Withers, "An Artificial Neural Network Integrated Circuit based on MNOS/CCD principles", AIP Conference Proceedings 151, "Neural Networks for Computing", pages 381 through 384, Snowbird, Utah, 1986. These all involve analog realizations of analog ANNs. Compared to digital realizations, they have the advantage of a significantly smaller implementation surface area. What is disadvantageous, however, is the high power consumption caused by the required resistance matrix, this power consumption opposing a realization of larger, programmable networks (with more than a few hundred neurons). The behavior and the properties of the modelings of ANNs hitherto proposed are preeminently investigated by simulation on vector computers, work stations or special processor fields. The fundamental disadvantage of this method is that the (space) parallelism inherent in the neural network is completely or partially lost in the processing of the information and, thus, the calculating time of the simulated network increases to such orders of magnitude, particularly for large neuron composites, that a speedy or quasi-real-time processing of the jobs cited above is impeded or becomes impossible.
A significant shortening of the calculating time and a far greater pattern throughput is obtained, by contrast, with hardware emulators. In contrast to hardware simulators, they contain an artificial neural network having a small size with whose assistance a larger network can be emulated.
In addition to a higher throughput, the advantage of a digital realization of an ANN is comprised in the greater freedom with which the user can select and set network-typical parameters (for example, discriminator functions).
The demands that users currently make of the degree of complexity (&gt;1000) for general neural networks currently exceed the possibilities of present VLSI integration technology.
In addition to these general, relatively large and complex neural networks, special solutions for specific jobs having smaller neural networks have already been proposed. These proposed, smaller networks are usually realized as complete solutions on a chip, being realized in analog technology with the disadvantages connected therewith.